Subclassification

Subclassification

Subclassification

Subclassification

The area of common support

• Subclassification is one way to adjust for differences using weighting, but there are some limitations here: what happens if I try to adjust for additional confounders? Or I try to group subjects into smaller age ranges?

The area of common support and ATT

• One way to mitigate the common support problem is to estimate the ATT instead of the ATE. If the missing data only appear in the treatment category, then we can still re-weight all of the non-zero treated units and recieve a valid ATT estimate:

Matching

Matching is an related method for addressing confounding. In the simplest matching models, we would try to find identify control units with the similar values of all confounders for every treated unit, then compare results.

Matching

In cases where we have more than one match for a single treatment unit, we can average over the matches:

\[ \text{ATT} = \frac{1}{N_T}\sum_{D_i=1} (Y_i -\left[\frac{1}{M} \sum^M_{m=1} Y_{j_m(1)} \right] \] … Where \(M\) represents the number of matching units for \(Y_i\)

Matching

Moreover, we can still estimate the ATE with this approach by finding counterfactuals for all treated and control units.

\[ {ATE} = \dfrac{1}{N} \sum_{i=1}^N (2D_i - 1) \bigg [ Y_i - \bigg ( \dfrac{1}{M} \sum_{m=1}^M Y_{j_m(i)} \bigg ) \bigg ] \]

Matching

…So how do we identify matching units? In most studies, “exact” matching is ideal, but unfeasible because of that common support requirement (especially when we want to adjust for continuous confounders like age)

Approximate Matching

For instance, we might use the Euclidean distance to compare vectors of covariates, then find the nearest match for each “point” (this is called nearest neighbor matching)

Approximate Matching

(Many nearest-neighbor matching studies will use the Mahalanobis distance instead, which is a metric that accounts for correlations between covariates)

Approximate Matching

The variations here are practically endless:

• For nearest neighbor matching, we might increase or decrease the number of matches permitted for each unit.

• We might also limit the distance between matched units so that, if nothing is close enough, we discard that match entirely.

• We might pair an approximate matching method with additional weighting methods to account for dissimilarities between pairs.

Propensity scores

Instead of matching on values themselves, propensity score methods either re-weight or match observations based on a predicted propensity to receive treatment:

• Estimate a model to predict being in the treatment group
• Then either:
  • Weight cases based on their inverse probability of treatment.
  • Match based on similar propensities (although this is now considered bad practice)

Propensity scores: inverse probability weighting

From: https://www.andrewheiss.com/blog/2024/03/21/demystifying-ate-att-atu/

Propensity scores: inverse probability of treatment

From: https://www.andrewheiss.com/blog/2024/03/21/demystifying-ate-att-atu/

Propensity scores

In practice propensity scores are often completely fine, but theoretically they can actually worsen bias if the treatment selection process isn’t modeled correctly. Other methods are more robust to this kind of mis-specification.

Coarsened Exact Matching

Coarsened exact matching works by:

Lalonde (1986)

• National Supported Work experiment: qualified applicants randomly assigned to either a treatment or control condition. Treatment group received a guaranteed job for 9-18 months with additional counseling and support

Lalonde (1986)

Lalonde (1986)

• Lalonde swapped the control group for a sample from the general population, and demonstrated that a similarly constructed study with observational data would fail to find the correct result, even after controlling for observed characteristics

Lalonde (1986)

(he tried a bunch of stuff, none of it worked)

Lalonde (1986)

Candidates for job training were unlike the general population. They weren’t even especially similar to people with similar income, education, age, race, etc.

Observed Imbalance

Imbalance

Control variables probably can’t fix this.

• large observed differences often imply large unobserved differences
• even if all confounding is observed and controlled, regression controls assume a linear relationship between the confounders and the treatment effects.

Imbalance

Applied for program \(\rightarrow\) Earnings

Applied for program \(\leftarrow\) Education \(\rightarrow\) Earnings

Imbalance

dataFunction<-function(){
  N<-1000
  education<-rpois(N, 3) # education
  D<-rbinom(N, 1, arm::invlogit(education)) # applied to job program
  Y_1<- 2 + education^2 +  rnorm(100) # potential outcome for training
  Y_0<- 0 + education^2 +  rnorm(100) # potential outcome for no training
  ATE<-mean(Y_1-Y_0) # the true effect 
  Y<-ifelse(D==1, Y_1, Y_0) # the observed outcomes 
  df<-tibble("income" =Y, Y_0=Y_0, Y_1=Y_1,
             'treated' = factor(D, labels=c("Control", "Treated")), 
             education)
  return(df)
}

set.seed(9999)
data<-dataFunction()
ggplot(data, aes(x=income,y=treated)) + 
  stat_halfeye() +
  theme_bw() +
  xlab("Potential Outcome if Y(1)")

mean(data$Y_1 - data$Y_0)

[1] 1.870983

Imbalance

Notably, since the education confounder has a non-linear effect on income, we don’t actually get a correct estimate even if we control for it:

Exact matching

Exact matching

Exact matching

Exact matching on education gets us close to the correct answer:

ate<-mean(data$Y_1 - data$Y_0)

matched<-matchit(treated ~ education ,data=data, 
                 method='exact',
                 estimand ='ATE'
                 )
mdat<-match.data(matched)

exact_match_fit <- lm(income ~ treated + education,
           data = mdat, weights = weights)

modelsummary(list('no controls'=ols, 
                  'ols + control' =ols_control, 
                  'exact matching' =exact_match_fit), 
             gof_map = c("nobs"),
             note = sprintf("actual ATE = %s", round(ate,digits=3))
             )

Exact matching

In repeated simulations, the exact matching estimates are unbiased and have only slightly more variance than the actual average treatment effect:

set.seed(999)
fun<-function(){
  df <- dataFunction() # re-using the data generating function
  ATE <- mean(df$Y_1 - df$Y_0)
  model<-lm(income ~ education + treated, data=df)
  matched<-matchit(treated ~ education ,data=df, method='exact' ,estimand='ATE')

  mdat<-match.data(matched)

  fit1 <- lm(income ~ treated ,
           data = mdat, weights = weights)
  res<-tibble('regression estimate' = coef(model)[3], 
                  #'matched' = mean(result$difference), 
              "matched estimate"=coef(fit1)[2],
              "Average Treatment Effect" = ATE
              )
return(res)

}

results<-replicate(500, fun(), simplify = F)|>
  bind_rows()|>
  pivot_longer(cols=everything())|>
  mutate(value = unlist(value))

Propensity scores

matched<-matchit(treated ~ education ,data=df, 
                 method ='nearest',
                 replace=TRUE,
                 distance='glm')
mdat<-match.data(matched)
#nrow(mdat)
psm_fit <- lm(income ~ treated + education , data=mdat, weights = weights)

modelsummary(list('ols + control' =ols_control, 
                  "exact matching"=exact_match_fit,
                  "psm" = psm_fit
                  ), 
             gof_map = c("nobs")
             )

Propensity scores

Coarsened Exact Matching

matched<-matchit(treated ~ education ,data=df, method='cem' ,
                 distance = 'glm', estimand = "ATE")
mdat<-match.data(matched)
cem_fit <- lm(income ~ treated + education , data=mdat, weights = weights)

modelsummary(list('ols + control' =ols_control, 
                  "exact matching"=exact_match_fit,
                  "psm" = psm_fit,
                  "cem" = cem_fit
                  ), 
             gof_map = c("nobs"),
             note = sprintf("actual ATE = %s", round(ate,digits=3))

             )

Standard errors

Methods like CEM will generally group variables together before weighting, so we would expect to see clustering within the strata, but we know how to deal with this!

 Estimate 2.5 % 97.5 %
     1.88  1.28   2.48

Term: treated
Type: response
Comparison: Treated - Control

Takeaways

• Matching can address some of the limitations of regression models and simulate experimental conditions.

• Main advantages over regression are:

  • No assumption of a linear relationship between confounders
  • More informally: allows researchers to separate out the treatment selection process from the treatment effect process.